On completion of this training sequence,
you will be able to understand the different quality assurance systems,
as well as the quality assurance criteria, used in the market.
Finally you will learn how to reach MARS,
or your capacity to produce accurate and rapid live subtitles.
This is the agenda of this presentation.
When dealing with quality assurance,
the four main recurrent criteria mentioned are accuracy, errors, speed and delay.
When it comes to accuracy,
various approaches to subtitling are mentioned,
each corresponding to specific strategies to apply,
that may be exclusive to that approach or transversal to more than one.
These approaches are the litteratim one,
or transcription of each sound of a given text;
verbatim, or transcription of each word;
sensatim, or the rendition of each meaning;
and signatim, or the rendition of each non-verbal sign.
Depending on the notion of accuracy requested by the client or the user,
our approach to subtitling will change.
The most common ones are the verbatim and the sensatim approach.
When assessing the quality of live subtitles,
the first thing that hearing people notice is that the subtitles are delayed
compared to the actual utterance of a given sentence.
This may sound quite banal to people in the field,
but this is something I always find myself explaining to clients.
There are basically two ways of counting delay,
and this depends on the approach used to subtitle a speech.
In case of verbatim subtitles,
delay is counted from the spoken word being uttered
to the same word appearing on screen in written form.
In case of sensatim subtitles or of interlingual subtitles,
delay is to be counted differently.
In particular, subtitles are counted from the concept being uttered by the speaker
to the same concept appearing on screen as written words.
If subtitles appear one word after the other as snake subtitles or roll-up subtitles,
delay is calculated between the average time of the spoken concept,
that is between the first and the last word being spoken,
and the average time of the corresponding written concept,
between the first and the last word being written down.
If subtitles appear as block subtitles,
delay is calculated between the last word of an uttered concept
to the time the block with the same concepts appears on screen.
Again, depending on the approach to adopt when subtitling,
various taxonomies to assess the quality of the subtitles exist.
When it comes to verbatim subtitles, two main models are used:
the traditional word error rate, or WER, and the NER model.
The NER model is widely used in the profession.
It consists in the following formula:
total number of words minus edition and recognition errors.
The total is divided by the number of subtitled words
and multiplied by 100.
Let’s take the sentence in the slide as an example.
«Well, you know, you have to try and put out a good performance.
I mean, yeah, it’s kinda stepping stone, ain’t it?».
Let’s pretend the subtitles get rid of features of orality and read:
«You must try to put out a good performance. It’s a stepping stone».
In this case, if we apply the NER model, the quality of the subtitles is 100%
because all missing or modified words are omitted or modified for the best.
When it comes to sensatim subtitles, however,
you need to use different models
to assess the quality of the subtitles.
Two main models are used:
the Idea-unit Rendition Assessment, or IRA,
and the Weighed Idea-unit Rendition Assessment, or WIRA.
These two models consider the meaning of each subtitle and not the words.
In the previous example,
the same quality rate would result from the application of both the IRA or WIRA.
However, the rationale is totally different.
Even a sentence expressing the same meaning with different words
and a different structure would result in 0 errors.
Regardless of the model used to assess errors,
errors are normally categorised in a similar way,
be they recognition errors or edition errors,
meaning errors caused by the machine or by a misuse of the machine.
Mistakes can be minor mistakes,
as in the case of George Bush being written with a lowercase B.
Such errors are rarely corrected by the live subtitler or by the live editor,
as they are easy to understand.
Standard errors are less easy to understand,
as they result in very different, though phonetically similar, words.
In some cases, you can still guess the original meaning
as in the case of the «European You Neon»,
and a subtitler may decide not to correct them.
In other cases, this is not an option as understanding is much harder,
as in the case of respeaker being recognized
as the surname Rees plus the noun peaker.
In this last case, readers cannot understand the meaning of the subtitle
but at least can tell the subtitle contains a mistake.
When readers cannot tell that the subtitles contain a mistake,
then we have major errors, or misleading mistakes,
as in the case of the sentence: «I'm a bespeaker»
instead of «I'm a respeaker».
In this case, the sentence contains something which is wrong.
Generally speaking,
acceptable verbatim subtitles are considered subtitles
containing less than 2% of mistakes.
Sensatim subtitles are considered of acceptable quality
when they contain less than 5% of mistakes.
In the Intersteno word championships of fast writing,
0.5% is the maximum number of mistakes allowed to competitors.
Speed is usually considered as the most important thing to assess
when assessing real-time subtitles.
For a subtitler, to train their speed is extremely important.
You can do so by doing two types of training:
a Text-to-Text training and a Speech-to-Text training.
Training with TAKI
is a very good way of training one's speed,
because it is a tool provided by Intersteno
that allows you to copy a written text at the speed you want.
The Speech-to-Text skills are better trained
with the Speech Capturing test, which is an Intersteno competition
that allows you to write down what you hear.
For a real-time intralingual subtitler,
however, it is not enough to be accurate and rapid.
It is fundamental to reach Mars!
Better, it is important to reach one’s MARS,
acronym for Most Accurate and Rapid Speech-to-text dictation rate.
In other words,
MARS is one’s capacity to produce accurate subtitles at a given speech rate.
A real time subtitler has to know his or her MARS as after this threshold,
subtitles are likely to be of less good quality.
After one’s MARS, professionals are to start thinking of exit strategies.
A minimum professional standard is 100 words per minute,
corresponding to 500 characters per minute,
not including the necessary voice commands to dictate punctuation.
This may sound a lot,
but real-life events can easily reach 200 words per minute.
This means it is extremely important
to constantly and consistently monitor one's MARS.
To calculate it, we have developed a tool that tells you
how many characters per minute or words per minute each of us can respeak
by keeping an acceptable error rate.
Go to reachmars.eu and enjoy!
In this video lecture, we have dealt with the last dictation skill,
which is at the core of quality assurance.
In particular, we have seen how to understand different quality assurance systems,
mainly verbatim and sensatim,
depending on criteria like accuracy, delay, errors and speed.
Finally, we have seen how to reach MARS,
or a respeaker's capacity to subtitle at a given speed
while keeping the error rate as low as possible.